Coded-array technique for obtaining depth and other position information of an observed object

ABSTRACT

A projection array is provided that comprises a plurality of discrete projection elements. An image array is obtained of a scene with the light projected onto it that is coded using the projection array. Correspondence information is determined for each element in the image array, where the correspondence information can be used to determine which of the plurality of elements in the projection array corresponds to a particular image element. The determination of correspondence information for each element in the image array can be made independently of correspondence information for other elements in the image array.

RELATED APPLICATIONS

This application claims benefit of priority to Provisional U.S. PatentApplication No. 60/390,232, filed Jun. 19, 2002, entitled “A Coded-GridDepth Sensor,” and naming Carlo Tomasi as an inventor. Theaforementioned priority application is hereby incorporated by referencein its entirety for all purposes.

FIELD OF THE INVENTION

The present invention relates to sensor systems that observe objects. Inparticular, the present invention relates to a method and apparatus forobtaining depth and other position information of an observed object.

BACKGROUND OF THE INVENTION

Three-dimensional sensor systems are increasingly being used in awide-array of applications. In one general application, athree-dimensional sensor system is used to determine the shape and/orfeatures of an object positioned in a scene of the sensor system's view.Many methods have been proposed for implementing sensor systems forsensing shapes and features in three-dimensions. Such methods typicallyrequire measurements of the time-of-flight or phase shift of lightreaching the scene from the sensor system. Other such methods usestructured-light or passive stereoscopic methods.

Structured-light methods have had success because of their superioraccuracy and resolution. Their most important drawback, on the otherhand, is that they require the projection of a moving beam or pattern oflight onto the observation scene. The need for a moving light patternimplies some mechanical moving component, which adds to the cost ordetracts from the reliability of the apparatus. Even more importantly,the need to scan the scene with a moving light beam, or to projectsequences of light patterns, means that one single depth map of thescene can only be captured over a period of time, often several seconds,and in some instances even minutes. As a result, structured-lightsystems are often poorly suited for real-time applications, in which theposition and configuration of the object of interest may change rapidly.

SUMMARY OF THE INVENTION

Embodiments of the invention provide for detecting characteristics of anobject. According to one embodiment, a projection array is obtained thatcorresponds to coded light projected onto a scene containing the object.The projection array may comprise a plurality of discrete projectionelements. An image array is obtained of the scene with the coded lightprojected onto it. A plurality of image elements are identified from theimage array. Correspondence information is determined for each imageelement that is identified from the image array, where thecorrespondence information can be used to determine which of theplurality of projection elements corresponds to that image element. Inone embodiment, the determination of correspondence information is madeindependently of other image elements in the image array for eachelement in the image array.

In another embodiment, a coded light pattern can be created on a scene,by creating a sequence of values that include a plurality ofsubsequences. Each subsequence may identify one or more elements of aprojection array. A value may be assigned to a characteristic in a setof optically distinguishable characteristics. Then, light may beprojected a pattern in order to create at least a portion of the patternon the scene. The sequence is encoded into the pattern based on thevalue assigned to each characteristic in the set.

Embodiments of the invention may be implemented on a system thatincludes a projector, a camera or other optical sensor, and a processor.Furthermore, embodiments of the invention may be embodied as a set ofinstructions that can be carried on a computer-readable medium. Suchinstructions may be executable by one or more processors to performsteps such as described above.

A computer-readable medium may include any medium that can holdinstructions that are retrievable and executable by a processor.Examples of a computer-readable medium include memory, optical memory,signal mediums that hold instructions, and external computer memory orstorage.

BRIEF DESCRIPTION OF THE DRAWINGS

Embodiments of the invention are illustrated by way of example, and notby way of limitation, in the figures of the accompanying drawings. Likereference numerals are intended to refer to similar elements amongdifferent figures.

FIG. 1 is a block diagram of an image correspondence system, under anembodiment of the invention.

FIG. 2 illustrates a method for determining correspondence between animage array and a projection array, such as provided with an imagecorrespondence system described in FIG. 1.

FIG. 3 is a diagram illustrating the concept of epipolar geometry, as itapplies to embodiments of the invention.

FIG. 4 illustrate row formation for different positions of a camera andprojector in an image correspondence system.

FIG. 5 illustrates principles for implementing triangulation fordetermining depth information, using correspondence information betweenan image array and a projection array.

FIG. 6 illustrates a method for obtaining depth information for anobserved object.

FIG. 7 illustrates a first error detection scheme for use with anembodiment of the invention.

FIG. 8 illustrates a second error detection scheme for use with anembodiment of the invention.

FIG. 9 illustrates a third error detection scheme that uses a parityvalue, under an embodiment of the invention.

FIG. 10 illustrates geometric symbols and shapes for encoding aprojection array, under embodiments of the invention.

FIG. 11 illustrates another set of geometric shapes for encoding aprojection array, under embodiments of the invention.

FIG. 12 illustrates an object encoded using a projection array withidentifiers such as shown in FIG. 10, under an embodiment of theinvention.

DETAILED DESCRIPTION OF THE INVENTION

Embodiments of the invention describe a coded-array technique forobtaining depth and other position information for an observed object.In the following description, for the purposes of explanation, numerousspecific details are set forth in order to provide a thoroughunderstanding of the present invention. It will be apparent, however,that the present invention may be practiced without these specificdetails. In other instances, well-known structures and devices are shownin block diagram form in order to avoid unnecessarily obscuring thepresent invention.

Overview

Embodiments of the invention provide for determining information aboutdiscrete points on an observed object by analyzing an image of theobject and referencing the image to known characteristics of lightprojected onto the object. In one embodiment, light is coded to providediscrete identifiable elements. The coded light is then projected ontoan object or scene. A camera or similar sensing apparatus obtains animage of the object. The elements of the coded light are captured in theimage. The captured elements in the image can be referenced to thediscrete elements provided by the coded projection in order to obtainthe information about the discrete points. One embodiment provides thatthe projection is coded so that each element detected from the image canbe associated with a corresponding element of an array through whichlight is projected. Furthermore, each element appearing in the image canbe associated with the corresponding element provided by the projectedlight independent of knowing the corresponding projection element of anyother image element.

Accordingly, embodiments of the invention provide a solution for what istermed in the vision perception and optical art as the “correspondenceproblem”. There have been past approaches that have contemplatedobtaining, for example, three-dimensional position information about anobserved object by projecting light as an array with a grid of knownpoints, and then obtaining an image of the array reflected off of theobject. But past approaches have had difficulty in determining whichelements in the projection array correspond to elements in the imagearray. This is the “correspondence problem”. If a processor was able todetermine which points of a projected pattern are present at individualpoints in the image of the pattern, that is, if the unit were able tosolve the so-called correspondence problem, then it would be a simplematter of triangulation geometry to determine the distance of eachprojected point from a camera that captures the image, assuming that therelative position, orientation, and optical parameters of both thecamera and the projector are known. Conventional structured-light shapesensors work according to this triangulation principle.

Lighting the scene with a-single beam at any one time usuallycircumvents the correspondence problem. One beam projects, say, a singleplane of light, which forms a curve when it intersects the surface ofthe object of interest. If the plane is vertical, and camera andprojector are side to side, then only at most one point of the curveappears in any given row of the camera's image, and the correspondenceproblem becomes trivial. In other words, time is used to disambiguateone curve from another in standard structured-light systems: rather thanprojecting several curves simultaneously, and facing the hardcorrespondence problem in which several luminous points appear in eachimage row, a standard structured-light system uses the time at which thecamera detects a single curve in order to tell which of the many curvesprojected by the light source is visible in which image. If the cameraand the light source are synchronized, then the correspondence problemcan be entirely avoided, and triangulation becomes possible.

There is a time and resource penalty associated with using a singleplane of light to view an object. A beam of structured light has to scanthe entire observed object. This takes time, and often makes theapproach impractical for many applications.

In contrast to past approaches, embodiments of the invention project theentire light pattern at once. The projected light pattern is designed tothat the correspondence problem can be readily solved.

One benefit of solving the correspondence problem is thatthree-dimensional position information about discrete points on theobserved object can be obtained from the correspondence solution. Otherinformation, including information that is dependent on a distancebetween the discrete points and a reference, may also be determined fromthe correspondence solution.

In the following description, FIG. 1 and the accompanying text describescomponents of a system that can be employed according to an embodimentof the invention. FIG. 2 and FIG. 3 and the accompanying text detailrespective methods for determining the column and row addresses orcoordinates of an image array in relation to a projection array. FIGS.4–5 and the accompanying text illustrate concepts behind determining therow coordinates of elements in an image array. FIG. 6 and theaccompanying text illustrate a method for obtaining depth informationusing addresses of elements in the image array. FIGS. 7–12 and theaccompanying text illustrate various specific implementation details andprovide examples.

General System and Methodology Description

FIG. 1 describes an image correspondence system, under an embodiment ofthe invention. As will be described, an image correspondence system maybe used to optically observe an object and determine three dimensionalfeatures, characteristics and other information of the object.

The components of such a system include a projector 10, a camera 120 anda processor 130. The projector 110 and the camera 120 are each alignedin a direction of a scene 102, where an object 105 is provided. In anembodiment, a known spatial or geometric relationship exists between thecamera 120 and the projector 110. Specifically, the camera 120 and theprojector 110 may be placed in a side-by-side orientation with respectto the scene 102. To simplify computations, an optical center of theprojector 110 and an optical center of the camera 120 may have paralleloptical axes.

The projector 110 projects an array 112 of coded light onto the scene102. Coded light refers to light that is structured or patterned so thatit can be represented as propagating from projector 110 as an array withidentifiable array elements. The projector 110 includes a light-emittingsource and some mechanism that causes light projecting from it to becoded. For example, projector 110 may include a bulb and a slide havinga pattern. When the light passes through the slide, the contents of theslide code the light. In one embodiment, light is projected as a gridwhen directed at the scene 102, so that grid points are thenidentifiable on the scene 102. The light may be coded so that the selectgrid points are identifiable, or otherwise distinguishable from eachother. The projection array 112 may correspond to a data structure thatrepresents a plane of light propagating from the projector 110 afterbeing structured into an array. In another embodiment, the projectionarray 112 may correspond to a tangible medium, such as a slide, thatincludes a grid pattern for producing the coded light structure.

In one embodiment, the scheme for coding the light provides that eachgrid point is assigned a value, and a sequence of values from a set ofcongruent grid points identifies one individual grid point amongst theset. The particular manner in which light can be coded to reflectsequence values for individual elements of an array is described in withFIGS. 7–12. In addition, the manner in which the light may be coded mayvary. In one embodiment, the light is structured so that each grid pointis assigned one symbol or geometric identifier from a set of two or moresuch identifiers. In another embodiment, grid points are colored to haveone color from a set of two or more colors. Still further, grid pointsmay be assigned a size from a set of two or more sizes. Other variationsand identifiers are possible. In the end, the coding scheme selectedassigns values to specific characteristics of light propagated form theprojector 110. In a binary scheme, two or three values (0,1 and parity)are utilized.

The projector 110 may be similar to a slide projector in applicationswhere electrical power is readily available. The coding may be providedby a slide containing a pattern, which in turn causes the projectedlight to have a pattern. In this case, the projection array 112 maycorrespond to a specific pattern formed on the slide. If it is necessaryto save energy, perhaps because the depth sensor is to work inconnection with a portable device, it is more effective to generate theprojection array 112 by diffracting a light source, such as a lightemitting diode (LED) or laser, into the desired pattern. The advantageof this configuration is its greater power efficiency, because all thelight from the source reaches the scene 102, rather than being reflectedback by the dark portions of a slide.

The pattern projected on the scene 102 may be visible, or it could beformed with invisible light when it is undesirable to reveal the array.For example, infrared light may be used when the infrared spectrum iseasier to distinguish from ambient light. The projected light may bepulsed in synchrony with a shutter in the camera 120, so as to optimizethe ratio of projected versus ambient light.

The camera 120 captures an image of the scene 102. When coded is lightis cast on the scene 102, the resulting image has a representation ofthe projection array 112. The image of the projection array 112 is atleast partially captured in image array 122. The camera 120 may includea lens (not shown) and a frame grabber which obtains data correspondingto image array 122. In one embodiment, the image array is a digitalrepresentation of the coded light on the scene 102 and the object 105.The depth of field of the camera should be sufficient to distinguish thedifferent types of code in the light coding emitted from projector 110.For example, in the case where the light coding corresponds to geometricshapes, the camera 120 is sufficient to distinguish amongst thedifferent symbols used. Enough depth of field in turn requires asufficient amount of light for a camera lens with a relatively smallaperture. In one embodiment, the camera 120 operates in black and whiteto identify geometric shapes or symbols that are of the order of onepixel. Alternatively, the camera 120 operates in color to identifycolored elements of the pattern provided by projector 110, where eacharray element occupies pixels of the order of less than ten pixels ofthe camera's pixel panel (not shown). In one embodiment, the camera 120is of a charged couple device (CCD) or complimentary metal-oxidesemiconductor (CMOS) design.

Embodiments of the invention provide that processor 135 receives datarepresenting image array 122 and projection array 112. Using techniquessuch as described in FIGS. 2 and 3, the processor 130 determines acorrespondence between an element of the image array 122 and an elementof the projection array 112. This correspondence is referred to ascorrespondence information 135. In one embodiment, processor 130determines a corresponding element in the projection array 112 for eachelement in the image array 122. Furthermore, the corresponding elementin the projection array 112 may be determined for any element in theimage array 122, independently of any correspondence between any otherelements in the image array 122 and the projection array 112. Thus, anembodiment such as described with FIG. 1 and elsewhere in thisapplication does not require a relationship to be known amongst elementsin the image array in order to determine the correspondence information.

Specific techniques and solutions to problems encountered in determiningthe correspondence information 135 are described below.

Obtaining Correspondence Information

FIG. 2 illustrates a method for determining correspondence between imagearray 122 and projection array 112. The correspondence between the imagearray 122 and the projection array 112 can be used to determine otherinformation, such as depth information for elements of the image array122. In this way, a three-dimensional image of the scene 102, includingthe object 105, can be obtained. Among other advantages, thethree-dimensional image is obtained quickly, with less demand on theprojector, camera and processing resources.

In step 210, an encoding scheme is developed for the projection array.In one embodiment, the encoding scheme comprises of sets of binarysequences that are non-repeating, so that each sequence in the set is aunique identifier for that set. A set of binary sequence may bedetermined as follows. The sequence s₁, . . . , s_(K) of K bits isnon-N-repeating if no subsequence of N bits ever repeats itself withinthe sequence. For instance, the following sequence of K=127 bits isnon-7-repeating:

-   -   1010101001100111011101001011000110111101010101011001001000111000010111110010101110011010001001111000101000011000001000000111111

It is easy to verify that no subsequence of seven contiguous bitsappears twice in the overall sequence set. This is the case even if thesequence is considered to be periodic, that is, its end is concatenatedwith its beginning.

There are many ways to generate such sequences, some of which areexplained in the book by S. W. Golomb, Shift Register Sequences, HoldenDay, San Francisco, 1967 (incorporated by reference herein for purposeof generating binary encoding sequences). A popular way is to use aso-called Linear-Feedback Shift Register (LFSR): a binary shift registerwith N bits b₀, . . . , b_(N−1), is initialized to any set of values,other than all zeros. At each of K iterations, a new bit is computedaccording to the following rule:

$b = {\sum\limits_{n = 0}^{N - 1}{p_{n}{b_{n}.}}}$

In this expression, the binary coefficients p₀, . . . , p_(N−1) arefixed and are determined appropriately so as to obtain the desiredsequence (see Golomb's book for a theory of how to do compute thesecoefficients). The register is then shifted by one position,

b_(N−1)=b_(N−2), . . . , b₁=b₀, and the newly computed bit is placed inthe first position of the register, b₀=b. The output sequence is thesequence of the newly computed bits.

For some values of N, the longest possible sequence of K=2^(N)−1 bitscan be generated in this way. For instance, all Fermat primes, that is,for prime numbers N that are themselves of the form N=2^(m)−1. As anexample, the number N=7=2³−1 is such a number, and was used in theexample above to yield a sequence of K=2^(N)−1=2² ³ −1=127 bits. Thepolynomial coefficients for the example werep₀=p₆=1, p₁=p₂=p₃=p₄=p₅=0.

Other methods exist for generating sequences of this type. Forapplications described herein, it is not necessary to obtain the longestpossible non-repeating sequence, but merely one that is long enough toavoid ambiguity.

In step 220, the code developed in step 210 is encoded into theprojection array. Discrete elements that comprise the projection arrayare assigned binary values that correspond to the coding schemedeveloped in the previous step. Each binary value (0 or 1) isrepresented by an optical identifier. In one embodiment, the opticalidentifier corresponds to a set of two symbols or shapes, where eachsymbol/shape in the set represents one of the two binary values. FIGS.10–11 illustrate some symbols or shapes that can be used as the opticalidentifiers. Another embodiment assigns colors to the set representingbinary values (e.g. red and blue).

In an embodiment, the projection array is coded to distinguish thecolumn address. As will be described with FIG. 3 and elsewhere in thisapplication, geometric or spatial relationships may be used to correlateeach row of the image array 122 to a corresponding row of the projectionarray 112. In an embodiment, the projection array 112 is dimensioned sothat the number of columns correspond to the largest non-repeatingsequence, which in the example provided above is 133 (127 plus 6 endelements). But other embodiments may use more than the longestnon-repeating sequence. For example, multiples of the non-repeatingsequence may be applied for each column of the projection array 112.

Step 230 provides that image array 122 is obtained from the scene. Theimage array 122 may be obtained by camera 120 when viewed over the scene102. Each element in the image array 122 may correspond to an image ofone of the optical identifiers cast with the projection array 112.

In step 240, correspondence information between independent elements ofthe image array 122 and the projection array 112 is determined throughthe encoding scheme. In an embodiment, the correspondence information isan address, provided in row and column coordinates, that identifies animage array element in terms of a projection array element. In anembodiment, the row address can be inferred from geometry alone, andmore specifically from the relative position and orientation of theprojector 110 that generates the pattern and of the camera 120 thatrecords its image. A description of this geometry is called the epipolargeometry of the projector-camera pair, which is known in the art.According to epipolar geometry, given the image position of a localpattern, there is only one row of the array that the pattern can belongto. Epipolar geometry determines which row this is.

For the column addresses, each image element is identified by asubsequence of the sequence provided with the projection array. Given anon-N-repeating sequence of length K, one can determine at least thecolumn address k of any of its subsequences S_(k) of length N, that is,the starting place of S_(k) within the sequence, in several ways. Forinstance, one could run the generation algorithm explained with step210, and count the number i of iterations needed until the contents ofthe shift register are equal to S_(k). The column address k is thenk=(i−N+1)mod K.

Alternatively, one can build a table of addresses ahead of time. Forinstance, Table 1 lists the addresses for the subsequences of length N=7within the sequence in the example above, cross-referenced against thecolumn coordinate.

TABLE 1 Addresses for the ^(K = 127) subsequences of length ^(N = 7) inthe sample sequence introduced previously: (0000001) 115 (0000010) 109(0000011) 116 (0000100) 110 (0000101) 61 (0000110) 103 (0000111) 117(0001000) 111 (0001001) 87 (0001010) 97 (0001011) 62 (0001100) 104(0001101) 28 (0001110) 55 (0001111) 118 (0010000) 112 (0010001) 52(0010010) 49 (0010011) 88 (0010100) 98 (0010101) 72 (0010110) 22(0010111) 63 (0011000) 105 (0011001) 7 (0011010) 81 (0011011) 29(0011100) 56 (0011101) 11 (0011110) 91 (0011111) 119 (0100000) 113(0100001) 101 (0100010) 85 (0100011) 53 (0100100) 50 (0100101) 20(0100110) 5 (0100111) 89 (0101000) 99 (0101001) 3 (0101010) 1 (0101011)73 (0101100) 23 (0101101) 41 (0101110) 75 (0101111) 64 (0110000) 106(0110001) 25 (0110010) 46 (0110011) 8 (0110100) 82 (0110101) 38(0110110) 43 (0110111) 30 (0111000) 57 (0111001) 77 (0111010) 16(0111011) 12 (0111100) 92 (0111101) 33 (0111110) 66 (0111111) 120(1000000) 114 (1000001) 108 (1000010) 60 (1000011) 102 (1000100) 86(1000101) 96 (1000110) 27 (1000111) 54 (1001000) 51 (1001001) 48(1001010) 71 (1001011) 21 (1001100) 6 (1001101) 80 (1001110) 10(1001111) 90 (1010000) 100 (1010001) 84 (1010010) 19 (1010011) 4(1010100) 2 (1010101) 0 (1010110) 40 (1010111) 74 (1011000) 24 (1011001)45 (1011010) 37 (1011011) 42 (1011100) 76 (1011101) 15 (1011110) 32(1011111) 65 (1100000) 107 (1100001) 59 (1100010) 95 (1100011) 26(1100100) 47 (1100101) 70 (1100110) 79 (1100111) 9 (1101000) 83(1101001) 18 (1101010) 126 (1101011) 39 (1101100) 44 (1101101) 36(1101110) 14 (1101111) 31 (1110000) 58 (1110001) 94 (1110010) 69(1110011) 78 (1110100) 17 (1110101) 125 (1110110) 35 (1110111) 13(1111000) 93 (1111001) 68 (1111010) 124 (1111011) 34 (1111100) 67(1111101) 123 (1111110) 122 (1111111) 121

For the present invention, a sequence embedded in the array of projectedsymbols must be long enough to avoid any ambiguity. Because of theepipolar geometry (which is described in more detail with FIG. 3), noambiguity is possible between epipolar lines: the epipolar line to whichan image point belongs determines the corresponding array epipolar lineuniquely. The position of an array point within a given epipolar line inthe image, on the other hand, depends on the geometry of the surfacebeing imaged. Ambiguity is avoided as soon as K exceeds either thenumber of symbols in a row or the disparity range (measured in number ofsymbols).

If this is the case, and if a subsequence of length at least N symbolsis observed within an image captured by the camera 120, the address ofthe subsequence can be determined.

FIG. 3 is a diagram illustrating the concept of epipolar geometry, as itapplies to embodiments of the invention. The projector optical center302 of the projector 110 (FIG. 1) and the camera optical center 304determine a line in space that is called the baseline 310. Consider theset of all planes that contain the baseline 310. These are called theepipolar planes 312 of the camera-projector pair. Some of the epipolarplanes 312 intersect the image area of the projector 110, as well as therectangle of the camera 120. For instance, if a standard slide projectorand a CMOS camera are used, some of the epipolar planes 312 willintersect the slide, and the same planes intersect the surface of theCMOS camera. The lines formed by these intersections are called epipolarlines in computer vision. The two lines formed by the intersection of asingle epipolar plane with the projector plane 322 and camera plane 324form an epipolar line pair. Note that all epipolar lines in theprojector plane 322 meet at a point, which by construction is theintersection of the baseline 310 with the projector plane 322. Thispoint is called the projector epipole 332. If the baseline 310 happensto be parallel to the projector plane 322, then the epipole is a pointat infinity on the projector plane. Similarly, the epipolar lines in thecamera plane 324 meet at the camera epipole 334, which is theintersection of the baseline 310 with the camera plane 324, and is apoint at infinity if the camera plane 324 is parallel to the baseline310.

FIGS. 4A and 4B shows what array rows look like in two different cases.In FIG. 4A, the camera 120 is adjacent to the projector 110, and theoptical axes of projector and camera are mutually parallel. In thiscase, both epipoles are points at infinity, and the array rows aremutually parallel lines. FIG. 4B shows the arrangement for when thecamera 120 is on the side of the projector 110, but slightly ahead ofit, so that the epipoles are finite points on both the projector and thesensor plane. The description provided in this document in generalassumes that the configuration of the system in FIG. 1 will resemble thesituation corresponding to FIG. 4A, since that configuration is easierthan the configuration provided by FIG. 4B. It should be clear, however,that any camera and projector arrangement is possible, as long as thearray rows on the projected pattern are oriented along the epipolarlines.

The importance of the epipolar geometry for embodiments described hereinis that any point on a given epipolar line in the projector plane 322must map onto the corresponding epipolar line in the camera plane 324.Thus, when the projector 110 projects an array of symbols onto scene102, the points in a given row of the array can only move within theirepipolar line in the image: no point can ever leave its epipolar line.

A curvilinear system of reference can be defined on the projectedpattern by placing the origin, say, at the top left corner of it. Columncoordinates increase along each epipolar line, and the row-coordinategridlines are point-wise orthogonal to the epipolar lines. FIG. 4A showsthat when the projector and camera are side-to-side with paralleloptical axes, the coordinates form a standard, orthogonal, Cartesiansystem.

A final, useful element from the geometry of a projector-camera pair isthe notion of disparity, which in turn requires the concept of areference surface. Imagine placing a plane anywhere in the scene, withthe only requirement that the same side of the plane be visible from theposition of both the projector 110 and the camera 120. When the array ofsymbols is projected onto the reference plane, the camera 120 forms animage of it. This image induces a system of reference onto the imageplane, in the sense that the array coordinates (r, c) of any point onthe array can be also assigned to the image point to which the arraypoint projects. Since the mapping between projector pattern points andimage points is one-to-one within the visible portion of the array, thisleads to a consistent assignment of coordinates to the image plane.

When a different surface replaces the reference plane, the image pointwith (array) coordinates (r,c) will generally be the projection of adifferent point of the array. As discussed above, and because of theepipolar geometry illustrated in FIG. 3, this other point has the samerow coordinate r, but possibly a different column coordinate c′. Thedisparity at that image point is defined as the discrepancy between thecolumn coordinates:d=c′−c.  (2)

The values of disparity depend on where the reference surface is placed.On the other hand, the disparity range, that is, the difference betweenthe smallest and greatest disparity that can be found at a given pointin the array and for a particular projector/camera arrangement dependsonly on the workspace in which the system is to operate. This is becausefor a particular array point the reference column coordinate c cancelswhen computing the disparity range:d _(max) −d _(min)=(c′ _(max) −c)−(c′ _(min) −c)=c′ _(max) −c′ _(min).

From the disparity at a particular point in an image, it is possible tocompute the three-dimensional position of the portion of the scene thatprojects to that point. The case of side-to-side camera 120 andprojector 110 is, again, a particularly simple one. The extension to thegeneral case is laborious but straightforward.

For side-to-side cameras, it is convenient to place the reference planefronto-parallel, and at infinite distance. Then, zero disparity isobtained when the surface is infinitely far away.

FIG. 5 illustrates that for a plane at infinity the projector-planepoint at array coordinates (r,c) projects onto the camera sensor atimage coordinates:

${\left( {\rho,\gamma} \right) = {\frac{f_{C}}{f_{P}}\left( {r,c} \right)}},$

where f_(P) is the focal length of the projector and f_(C) is that ofthe camera. Also, comparison of similar triangles in the figure readilyyields the desired relationship between disparity and depth Z, measuredalong the common direction of the two optical axes, for a point on anarbitrary surface:

$Z = {f_{P}{\frac{B}{d}.}}$

In this expression, B is the length of the baseline, and

$d = {c - {\frac{f_{P}}{f_{C}}\gamma}}$

is disparity.

To use this formula, it is necessary to know c, that is, the columnaddress of the array point that happens to be visible at image position(ρ,γ). This is determined using methods such as described with FIG. 2.In this way, the depth of a particular point on the observed object 105may be determined from the disparity (which requires knowing the columnaddresses and correspondences between the image and the projectionarray), and the spatial/geometric relationship between the projector 110and camera 120.

Obtaining Depth From Correspondence

FIG. 6 illustrates a method for obtaining depth information for anobserved object, using correspondence information and knownrelationships between the projector 110 and the camera 120. In step 610,sequence sets are generated that identify individual column addresses inthe projection array 112. For example, a non-N-repeating sequence may begenerated for elements of the projection array 112, as described withFIG. 2.

Step 620 provides that a sequence comprising the generated sequence setsare encored into the array of light emitted from the projector. In oneembodiment, the non-repeating symbol sequences are embedded into a lightpattern or structure projected onto the scene, so that subsequentexamination of one of the sequences reveals the column position of thatsubsequence within its row of the array. In order to perform this step,three sub-steps may need to be performed: (i) the one-dimensionalsequences are embedded into a two-dimensional array so that small imagepatches contain all the necessary information for address computation;(ii) an error-detection code is superimposed onto the sequence, in orderto reveal missing or misread symbols; and (iii) binary symbols areencoded in a way that makes the symbols easily readable in an image of ascene lit with the array. Each of the subsets are described in greaterdetail below.

Step 630 provides that the coded array of light is cast onto the scene102. For example, a light may be emitted through a slide that containsthe sequence sets encoded into a grid provided on the slide.

In step 640, an image of the scene 102 is captured using the camera 120.The captured image is assumed to include an image of at least a portionof the projection array 112, which was used in casting the light ontothe scene 102. The image of the projection array 112, however, may bedistorted in that not all of the projection array may have beencaptured, and surfaces of the scene 102 may have altered portions of theprojection array 112 as captured in the image.

In step 650, values are determined for elements in the array captured bythe image. For example, the image of the scene with the structured orpatterned light may be interpreted into a grid containing valuesidentified from patterned light reflecting off of the scene 102. Thus,the contents of the array captured in the image correspond to at least aportion of the projection array 112. The resulting array of values maycorrespond to image array 122. For example, the sequence generated instep 610 may result in a string of binary values for the image array122.

Step 660 provides that the row address for corresponding individualelements of the image array 112 to one of the elements of the projectionarray 122 are determined using known spatial and geometric relationshipsbetween the camera 120 and the projector 110. As described with FIG. 3,for example, rows in the image array 122 may have corresponding rows inthe projection array 112.

Step 670 provides that the sequence sets are used to determine thecolumn address for corresponding individual elements of the image arrayto one of the projection array elements. For example, sequence setsgenerated in step 610 may be decoded using a technique such as describedwith FIG. 2.

In step 680, depth information is determined for elements in the imagearray 112 using the row and column correspondence determined in step 660and step 670. In one embodiment, triangulation techniques such asdescribed in FIG. 5 may be employed to determine how far a region on thesurface represented by an element of the image array 122 is from thecamera 120. This distance may be determined by determining how thecolumn coordinate of the image element differs from its correspondingprojector element, as well as knowing a spatial relationship between thecamera 120 and the projector 110.

A method such as described in FIG. 6 may be used to obtain a depth mapof object 105. The depth map may correspond to an array that assigns adepth value to every element of the image array 122. The depth map maybe determined at relatively quick time intervals, such as several timesper second. The use of depth maps has various applications describedbelow, such as facial recognition.

Embedding Binary Sequences into Projection Array

One of the subsets in step 620 of FIG. 6 is to encode a sequencecomprising the subsets into the projection array 112. In general, manyarrangements for encoding a sequence into an array, such as theprojection array 112, are possible. The simplest makes all rows of thearray the same, with the same non-repeating sequence in each row:

$\begin{matrix}s_{0} & s_{1} & s_{2} & s_{3} \\s_{0} & s_{1} & s_{2} & s_{3} \\s_{0} & s_{1} & s_{2} & s_{3} \\s_{0} & s_{1} & s_{2} & ⋰\end{matrix}$

Each column of the array would then encode one bit of the non-repeatingsequence. This scheme is best suited for small values of N, the lengthof the non-repeating subsequence. For values of N greater than, say, 4or 5, the address calculator would have to examine a long stretch of anarray row to determine the address of the subsequence. The probabilitythat an error is detected in such a stretch is relatively high,especially in the presence of numerous occlusions in the scene. As aresult, many missing depth values would result as a consequence.

A better embedding scheme for greater values of N is to staggersubsequences in consecutive rows by the square root of N, rounded up tothe next integer. For instance, for N=7, the subsequences are embeddedas follows:

$\begin{matrix}s_{0} & s_{1} & s_{2} & s_{3} \\s_{3} & s_{4} & s_{5} & s_{6} \\s_{6} & s_{7} & s_{8} & s_{9} \\s_{9} & s_{10} & s_{11} & ⋰\end{matrix}$

In this arrangement, each row contains a Linear-Feedback Shift Register(LFSR) sequence, staggered by three elements relative to the previousrow. Such a sequence has the property that for a particular primeinteger N the sequence is K=2^(N)−1 bits long, and no subsequence of Nbits is ever repeated within the sequence. Then, a complete address canbe computed whenever at least seven consecutive symbols can be read.This is achieved with windows of size 1×7, 2×4, or 3×3 symbols, therebysubstantially increasing the likelihood that at least one of thesewindows is error-free.

Detecting Missing or Misread Elements of the Projection Array

Coding the light from the projector 110 requires assigning an opticallydistinguishable attribute to each element of the projection array 112.For purpose of description, each element of the projection array 112 issaid to be assigned a symbol representing a binary value. Whenprojecting an array of symbols, it is possible, and even likely, thatnot all of the symbols will be projected or read correctly. For example,some of the following problems may arise: (i) not all projected symbolsmay be within the field of view of the camera 120, so that it may bedifficult to determine which is the first (or last) symbol that isvisible in the image;(ii) symbols may project on very dark parts of thesurface, and become consequently invisible to the camera; (iii) symbolsmay project onto surface patches that are not visible from the camera'sposition (such symbols may be referred to as occluded symbols); and (iv)the shape of a symbol may become distorted, blurred, or otherwisecorrupted beyond recognition.

To address these difficulties, the array of symbols projected onto thescene 102 may be endowed with two properties. The first property ensuresthat examination of a small pattern from the array allows computing theposition or address of the pattern within the array itself. This may beachieved by designing the symbols in the array so that groups of symbolswithin the same row in the array are never repeated.

The second property allows checking whether the small pattern inquestion contains an unbroken sequence of contiguous symbols. Thisavoids problems that may be caused because of missing or misinterpretedsymbols that interfere with the pattern address calculation mentionedabove. This second property is enforced by the insertion of redundantsymbols that implement an error-detecting code.

If these two properties are satisfied, a group of contiguous symbols inthe image contains, so to speak, its own address: the arrangement ofsymbols in the group is unique to that particular group in that row ofthe projected array, and it cannot be confused with any other group.Correspondence between projected and imaged group can therefore beestablished. Standard triangulation, such as described in FIG. 5, maythen be applied between individual points within the group of symbols inorder to compute a high-resolution depth map of the scene 102.

One example of how a pattern of non-repeating symbols could be producedis as follows. Let s_(n) be a bit in a so-called binary maximal-lengthLFSR. To fix the ideas, let N=7, and write out the sequence s₁, . . .,s₁₂₇ of K=2^(N)−1=127 bits. No subsequence of N=7 bits ever repeatswithin these 127 bits. One way to write these bits into the projectedpattern is to encode zeros with one type of symbol, and ones withanother type. Several arrangements of the sequence into the array arepossible.

Error detection can be used to reveal missing or misread symbols. Thisis a known technique in communication and storage systems. The simplesttechnique is to add a parity bit after every M bits in the sequence.This bit is the modulo-2 sum of the preceding M bits. A single error,occurring either in the M bits or in the parity bits, can be detected atreading time. To this end, the parity bit is recomputed and checkedagainst the one read from the image. If the two bits differ, an oddnumber of errors must have occurred. Other techniques, such as CyclicRedundancy Check (CRC) codes, may be used as well, but require largerblocks of symbols to be considered at once. Several error-detectingcodes are discussed in W. W. Peterson and D. T. Brown, “Cyclic Codes forError Detection,” Proc. IRE, January 1961, pp. 228–235.

The most frequent cause of missing symbols is occlusion, where symbolsprojected onto part of the scene are hidden from the camera 120 by otherparts of the scene 102, or, conversely, visible parts of the scene arenot reachable from the projector because of other intervening objects.If these occlusions, or missing symbols caused by other problems, wereto be ignored, it would be possible to read disconnected fragments of asequence without realizing the lack of continuity. This would in turnyield a wrong subsequence address, incorrect disparity, and erroneousdepth value.

One simple method to detect when symbols are missing or misread is toexploit the fact that adjacent rows contain the same subsequence,perhaps staggered as discussed in the previous section. For instance,one could check that symbols in the proper places coincide, asillustrated in FIG. 7.

Similarly, one can check coincidences in the windows of size 1×7, 2×4,or 3×3 symbols mentioned in the previous section, as illustrated in FIG.8.

These checks, however, do not detect errors in the symbols in-between,unless the same checks are repeated for overlapping windows.

A more systematic alternative is to add symbols whose sole purpose is todetect errors. This is standard practice in the field of so-callederror-detecting coding theory (see for instance A. S. Tanenbaum,Computer Networks, Prentice-Hall, 1996). A wide variety of errordetection codes are used in many applications, such as checking theintegrity of data on computer disk drives or at the receiver end of anoisy communication channel. Any of these coding schemes (parity bits,cyclic redundancy check codes, and others) can be used in ourapplication as well. In the following, a scheme is presented that lendsitself to effective embedding into the arrays used in the presentinvention.

Continuing the example with N=7, parity bits can be added on the marginsof every 3×3 sub-array, as shown in FIG. 9. In FIG. 9, b₀ is the parity(i.e., the modulo-2 sum) of the three bits to its left, that is,b₀=s₀⊕s₁⊕s₂, and similar equations hold for b₁,b₂,b₆,b₇,b₈. Bits at thebottom of each column are parity bits for the three bits above them:b₃=s₀⊕s₃⊕s₆, and similar equations for b₄,b₅,b₉,b₁₀,b₁₁. The specialsymbol denoted with d is different from the two symbols used to denote 0and 1, and is used to delimit 4×4 blocks, so that parity bits can bedistinguished from sequence bits.

Other error-detection schemes are possible as well. For instance, thecolumn-parity bits (or the row-parity bits) could be skipped, differentblock sizes could be used, or functions other than parity could becomputed for the additional bits. In general, a block of symbols is saidto be addressable if its symbols satisfy all the error-detection checks,and if they are sufficient to determine the column address of the blockwithin the array.

Computation of Depth at High Resolution

Once the validity of the bits encoded by a block of symbols within thearray has been established, the sequence bits within that block can beused to determine the column address of the block within the array. Therow address is determined from the epipolar geometry, as discussedearlier. These computations yield the coarse, symbol-level disparity ofthe block of interest, from which a crude estimate of depth can be made.A more precise determination of depth requires matching of individualarray points and image pixels at a finer resolution.

Since the system at this point knows which symbol is which, this finerresolution can be achieved by computing disparities for each point onthe boundary of each symbol. In each epipolar line in the image, astandard edge detection algorithm can be used to find the symbolboundaries, and brightness interpolation methods can compute thelocations of these boundaries to better than pixel resolution. Methodsfor edge detection and sub-pixel interpolation are known in the art, andcan be found in most books on computer vision or image processing (seefor instance J. C. Russ, The Image Processing Handbook, CRC Press,1999).

Depth of Non-Addressable Symbols

In some cases, a surface patch may be found to be non-addressable,because the bit block that covers it fails some of the error-detectionchecks, or because the patch contains too few bits. Such a patch is notnecessarily lost to the depth sensor. For instance, a neighboring blockmay be addressable. If this is the case, the subsequence of thenon-addressable block may be narrowed down to a small number ofpossibilities. Given this restriction, it is often possible to determinean address from fewer than N bits.

Identifiers for use with Projection Array

According to an embodiment, symbols, shapes and other opticalidentifiers that are projected onto scene 102 as part of the projectionarray 112 may require some of the following characteristics.

First, there needs to exist a sufficient number of distinct identifiersto satisfy coding requirements. In binary coding, for example, two orthree distinct symbols are needed. Two identifiers are needed if onlynon-repeating sequences are encoded, and three if a delimiter symbol isused, as shown with FIG. 9.

Second, the identifiers should be readily distinguishable, particularlythrough the camera 120. For example, if symbols are used as theidentifiers, the symbols should have high contrast to one another, sothat each symbol's boundaries can be found accurately. Furthermore, thesymbols may be bright, or brightly shaded, so as to be visible even whenprojected onto dark surfaces.

Third, distinct symbols must be clearly distinguishable in the image,even when they are projected onto slanted surfaces.

There are no fundamental restrictions as to the shapes of the symbolsused as identifiers. The main design criterion is that different symbolsshould be distinguishable. However, the brightness and contrastrequirements above suggest projecting binary images (that is, imageswith only two levels of brightness, black or white). If a symbol isthought of as being drawn into an ideal rectangle whose size is equal tothe pitch of the array of symbols, it is desirable for the symbol tocover about half of the area of the rectangle. This maximizes contrasteven with blurred images, and at the same time makes the distribution ofsymbol boundary points relatively uniform, leading to depth maps thatare more evenly populated with data points.

The desire to have as many addressable blocks as possible imposes anupper limit on the size of symbols to use, since large blocks are morelikely to straddle occlusions. However, this size limit depends on theapplication. For instance, a person's face contains few if any depthdiscontinuities, and the symbols can be relatively large (roughlyspeaking, a single addressable block could cover the entire face). Onthe other hand, to handle, say, hands appropriately, symbols would haveto be small enough that an addressable block is likely to fit entirelyon a finger.

The size of symbols in turn affects the required camera resolution: onesymbol must cover enough pixels in the image that distinct symbols areeasily distinguishable in the presence of complex surfaces, distortion,blurring, and image noise. FIG. 10 shows some examples of symbol shapes.In FIG. 10, any two of the shapes provided may be used to represent oneof two binary values. Reliance on handedness (left) or shape differences(other symbol pairs) guarantees that different symbols can bedistinguished even when they are projected onto slanted surfaces.

FIG. 11 shows another embodiment where the identifiers are blocks ofdifferent sizes. For example, a tall block may be assigned to representa first binary value, and a shorter block may be assigned to represent asecond binary value. The resulting image array may be analyzed todetermine boundaries corresponding to each of the rectangular shapes.

FIG. 12 illustrates a face in which a projection array is cast accordingto a coding scheme. In FIG. 12, the identifiers correspond to the shapesdescribed with FIG. 11.

Calibration

With some embodiments, a calibration procedure may be performed todetermine the image location of each identifier in the projection arrayfor a scene containing only the reference plane. The map of all suchimage locations is called a reference map. This map matches points onthe projector plane with points in the camera sensor plane, and istherefore a homography. It is known in the art that such a homographycan be computed from a small number of points (see for instance RichardHartley and Andrew Zisserman, Multiple View Geometry in Computer Vision,Cambridge University Press, Cambridge, UK, 2000).

With some embodiments, a calibration procedure may be performed todetermine the image location of each identifier in the projection arrayfor a scene containing only the reference plane. The map of all suchimage locations a reference map. This map matches points on theprojector plane with points in the camera sensor plane, and is thereforea homography. It is known in the art that such a homography can becomputed from a small number of points (see for instance Richard Hartleyand Andrew Zisserman, Multiple View Geometry in Computer Vision,Cambridge University Press, Cambridge, UK, 2000).

Applications

Embodiments of the invention have several applications. For example, inone embodiment, an airbag deployment system may be provided in anautomobile that can be tuned to the shape and size of the passenger.Information about the shape and/or size of the passenger may bedetermined from a depth map of the person's face or body. Such a depthmap may be obtained by a sensor system that operates in a mannerdescribed with embodiments of the invention.

As another application, a facial recognition system may be provided thatuses information about the facial shape and features of a person. Suchinformation may also be obtained from a depth map, using sensors such asdescribed for use with embodiments of the invention. Given sufficientdepth information, the facial recognition system may be enhanced in itsability to separate a person's head from the background, or in itsability to provide three-dimensional head-shape parameters that can easeidentification of the person.

Another application that can incorporate embodiments of the invention isa gesture recognition system. Using a sensor system such as described,depth information can be obtained regarding the pose, posture, gestureand/or movement of a person's extremities or other body parts. In thisway, the user's hands, arms, and general upper body gestures can beinterpreted as commands for some device. For example, the user'sgestures may be interpreted as commands for a remote control or avirtual keyboard. Alternatively, a speech translation mechanism may beimplemented that translates gestures corresponding to sign-language intospeech.

Still another application for an embodiment of the invention is athree-dimensional sensor system that can separate foreground image frombackground image in a video stream. This allows the background image tobe replaced or otherwise manipulated for visual effects.

Other applications may involve determining the shape of objects. Forexample, a virtual-reality system can be populated with virtual,three-dimensional objects, whose shape can be scanned by athree-dimensional sensor, such as provided herein. Thus, the objects ofthe virtual system may be designed through use of sensor systems thatobserve the physical world. Another application is to determine theshape of an object, such as a mechanical part or an art object. Thecalculated shape may be more readily represented by data, which in turnmay be transmitted over a communication channel to a remote location.

On a larger scale, three-dimensional shapes can be usefully capturedusing depth information or three-dimensional systems, configuredaccording to an embodiment of the invention. For example, sensors suchas described may be used to capture an architectural survey, measurevenues and locations for a variety of purposes, monitor traffic in orderto assist drivers.

CONCLUSION

In the foregoing specification, the invention has been described withreference to specific embodiments thereof. It will, however, be evidentthat various modifications and changes may be made thereto withoutdeparting from the broader spirit and scope of the invention. Thespecification and drawings are, accordingly, to be regarded in anillustrative rather than a restrictive sense.

1. A method for detecting characteristics of an object in a scene, themethod comprising: projecting a projection array, comprising a pluralityof projection elements arranged in rows and columns, to project ontosaid scene and object coded light creating a pattern of said projectionelements having a detectable characteristic that identifies a column insaid projection array and belongs to a characteristic set of at leasttwo characteristics, said pattern of projection elements being createdfrom said projection array and defining a repeated pattern of staggeredsaid columns; capturing an image of said scene with said coded lightprojected onto said scene; identifying a plurality of image elementsforming an image array from a captured said image; and determining foreach image element in a captured said image correspondence informationuseable to determine which projection element in said plurality ofprojection elements corresponds to that image element, whereindetermining correspondence information for each element is performedindependently of knowing correspondence information for any other imageelement.
 2. The method of claim 1, further comprising determiningposition information for each identified image element based at least inpart on correspondence information for said identified image element. 3.The method of claim 1, further comprising: providing a protector toprotect said projection array; providing a camera to capture said imagearray; and determining position information for each identified imageelement based at least in part on correspondence information for saididentified image element and on a position of said projector relative tosaid camera.
 4. The method of claim 1, wherein determiningcorrespondence information includes determining a row coordinate and acolumn coordinate in said image array for at least one said imageelement based at least in part on a row coordinate and a columncoordinate in said projection array.
 5. The method of claim 1, whereindetermining correspondence information includes: determining a rowcoordinate and a column coordinate in said image array for at least onesaid image element; and for each at least one said image element,determining a row coordinate and a column coordinate for an element insaid projection array corresponding to said image element.
 6. The methodof claim 1, wherein determining correspondence information is performedabsent prior determination of correspondence information for acalibration element in said plurality of image elements designated ascalibrating correspondence information for all other image elements insaid plurality of image elements.
 7. The method of claim 1, whereindetermining correspondence information is performed absent priordetermination of correspondence information for an element in saidplurality of image elements designated as being a first element in asequence of elements that correspond substantially to an entire row ofelements in said projection array.
 8. The method of claim 1, whereindetermining correspondence information includes identifying missingimage elements by identifying at least one projection element lacking acorresponding image element in said image array.
 9. The method of claim8, further comprising compensating for missing image elements by usingimage elements congruent to said missing image elements.
 10. The methodof claim 1, wherein projecting a projection array is carried out with aprojector and includes identifying data representing a tangible mediumused to pattern light emitted from said projector.
 11. The method ofclaim 1, further including: providing a projector to protect saidprojection array; and identifying a projection array includesidentifying data representing a diffraction pattern for structuringlight emitted from said projector.
 12. The method of claim 1, whereineach characteristic in said characteristic set has a characteristicselected from a group consisting of (a) a geometric characteristicdistinguishable from an other geometric characteristic in said set, (b)color from another color in said set, and (c) a shape distinguishablefrom another shape in said set.
 13. The method of claim 1, furthercomprising: determining a sequence of values that can identify eachprojection element in said projection array; assigning a sequence valueto each characteristic in said characteristic set; and whereinprojecting coded light includes projecting each projection element inprojection array as a characteristic in said characteristic set so thatsaid sequence can be at least partially reflected in said pattern basedon sequence value assigned to each characteristic.
 14. The method ofclaim 13, wherein assigning a sequence value includes assigning in amanner selected from a group consisting of (a) assigning a binary value,(b) assigning non-repeating binary values, (c) assigning using anon-N-repeating bit assignment formula, and (d) assigning a sequencesubset to individual column coordinates so as to correspond elements insaid projection array with elements in said image array.
 15. The methodof claim 1, wherein projecting coded light includes creating saidpattern so that detectable characteristics identify columns in saidprojection array and so that each row in said projection array has asame set of columns.
 16. The method of claim 1, further comprising usingsaid staggered arrangement to determine which projection elements insaid projection array lack a corresponding image element.
 17. The methodof claim 1, further comprising identifying rows in said image array thatlack a same column to determine which projection elements in saidprojection array lack a corresponding image element.
 18. The method ofclaim 1, wherein at least one method step is performed on at least oneprocessor executing instructions stored on a computer-readable medium.19. A system to detect characteristics of an object in a scene, thesystem comprising: means for projecting a projection array, comprising aplurality of projection elements arranged in rows and columns, toproject onto said scene and object coded light creating a pattern ofsaid projecting elements having a detectable characteristic thatidentifies a column in said position array and belongs to acharacteristic set of at least two characteristics, said pattern ofprojection elements being created from said projection array anddefining a repeated pattern of staggered said columns; means forcapturing an image of said scene with said coded light projected ontosaid scene; means for identifying a plurality of image elements formingan image array from a captured said image; and means for determining foreach image element in a captured said image information useable todetermine which projection element in said plurality of protect elementscorresponds to an image element, wherein determining correspondenceinformation for each element is performed independently of knowingcorrespondence information for any other image element.
 20. The systemof claim 19, wherein said means for projecting includes a projector, andidentifies data representing a tangible medium used to pattern lightemitted from said projector.
 21. The system of claim 19, wherein saidmeans for projecting includes a projector and includes data representinga diffraction pattern for structuring light emitted by said projector.22. The system of claim 19, a characteristic in said characteristic sethas a characteristic selected from a group consisting of (a) a geometriccharacteristic distinguishable from an other geometric characteristic insaid set, (b) a color from another color in said set, and (c) a shapedistinguishable from an other shape in said set.
 23. The system of claim19, further including: means for determining a sequence of values thatcan identify each projection element in said projection array; means forassigning a sequence value to each characteristic in said characteristicset; wherein said means for projecting projects coded light and projectseach projection element in said projection array as a characteristic insaid characteristic set so that said sequence can be at least partiallyreflected in said pattern based on sequence value assigned to eachcharacteristic.
 24. The system of claim 19, wherein a sequence value isassigned in a manner selected from a group consisting of (a) assigning abinary value, (b) assigning non-repeating binary values, (c) assigningusing a non-N-repeating bit assignment formula, and (d) assigning asequence subset to individual column coordinates so as to correspondelements in said projection array with elements in said image array. 25.The system of claim 19, wherein said staggered arrangement is used indetermining which projection elements in said projection array lack acorresponding image element.
 26. The system of claim 19, wherein saidmeans for projecting projects coded light and creates said pattern sothat detectable characteristics identify columns in said projectionarray and so that each row in said projection array has a same set ofcolumns.
 27. The system of claim 19, wherein said means for determiningdetermines position information for each identified image element basedat least in part on correspondence information for said identified imageelement.
 28. The system of claim 19, wherein: said means for identifyinga projection array includes a projector; said means for obtaining animage includes a camera; and said means for determining determinesposition information for each identified image element based at least inpart on correspondence information for said identified image element andon a position of a said protector relative to said camera.
 29. Thesystem of claim 19, wherein said means for determining determines a rowcoordinate and a column coordinate in said image array for at least onesaid image element based at least in part on a row coordinate and acolumn coordinate in said projection array.
 30. The system of claim 19,wherein said means for determining determines a row coordinate and acolumn coordinate in said image array for at least one said imageelement; and for each at least one said image element, determines a rowcoordinate and a column coordinate for an element in said projectionarray corresponding to said image element.
 31. The system of claim 19,wherein said means for determining functions absent prior determinationof correspondence information for a calibration element in saidplurality of image elements designated as calibrating correspondenceinformation for all other image elements in said plurality of imageelements.
 32. The system of claim 19, wherein said means for determiningfunctions without prior determination of correspondence information foran element in said plurality of image elements designated as being afirst element in a sequence of elements that correspond substantially toan entire row of elements in said projection array.
 33. The system ofclaim 19, wherein said means for determining identifies missing imageelements by identifying at least one projection element lacking acorresponding image element in said image array.
 34. The system of claim33, wherein said means for determining compensates for missing imageelements by using image elements congruent to said missing imageelements.
 35. The system of claim 19, further including identifying rowsin said image array that lack a same column to determine whichprojection elements in said projection array lack a corresponding imageelement.
 36. The system of claim 19, further including identifying rowsin said image array that lack a same column to determine whichprojection elements in said projection array lack a corresponding imageelement.
 37. A system to detect characteristics of an object in a scene,the system comprising: a projector to project a projection array,comprising a plurality of projection elements arranged in rows andcolumns, and to project onto said scene and object coded light creatinga pattern of said projecting elements having a detectable characteristicthat identifies a column in said position array and belongs to acharacteristic set of at least two characteristics, said pattern ofprojection elements being created from said projection array anddefining a repeated pattern of staggered said columns; a camera tocapture an image of said scene with said coded light projected onto saidscene; means for identifying a plurality of image elements forming animage array from a captured said image; and means for determining foreach image element in a captured said image information useable todetermine which projection element in said plurality of project elementscorresponds to an image element, wherein determining correspondenceinformation for each element is performed independently of knowingcorrespondence information for any other image element.